Papers with low-entropy ensemble distribution
Entropy-Based Decoding for Retrieval-Augmented Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and intrinsic knowledge sources. |
| Approach: | They propose a entropy-based document-parallel ensemble decoding method that prioritizes low-entropies from retrieved documents and incorporates a contrastive decoding mechanism that contrasts the obtained low- and high-entropic ensemble distributions with the high-end internal knowledge across layers. |
| Outcome: | The proposed method improves on open-domain question answering datasets and shows that it is highly efficient. |